Move public nightlies to PyPI and keep local installs stable#2220
Open
charlesbluca wants to merge 9 commits into
Open
Move public nightlies to PyPI and keep local installs stable#2220charlesbluca wants to merge 9 commits into
charlesbluca wants to merge 9 commits into
Conversation
charlesbluca
changed the title
Contributor
Greptile SummaryThis PR migrates public nightly publishing for
|
| Filename | Overview |
|---|---|
| .github/workflows/pypi-nightly-publish.yml | Drops TestPyPI routing, adds workflow-level permissions: contents: read, and migrates token handling to TWINE_PASSWORD env var — both previously flagged security findings are resolved here. |
| .github/workflows/huggingface-nightly.yml | Removes TestPyPI branch (hard-codes PyPI URLs), adds nightly_base_version matrix field for three non-OCR packages, and simplifies upload flag logic; already carried permissions: contents: read at workflow level. |
| ci/scripts/nightly_build_publish.py | Updates CLI defaults (--repository-url, --token-env) to PyPI; _twine_upload already passes the token via TWINE_PASSWORD env var rather than as a CLI arg — no substantive logic change. |
| nemo_retriever/pyproject.toml | Converts pinned Nemotron versions in [local] extra to stable semver ranges (>=x,<next-major), adds PEP 735 [dependency-groups] dev section with nightly exclusion constraints, and removes the test-pypi uv index. |
| nemo_retriever/tests/test_ci_workflows.py | Adds three new tests enforcing that public nightly workflows target only PyPI, carry least-privilege permissions, and use TWINE_PASSWORD env var — directly codifying the security properties introduced by this PR. |
| nemo_retriever/tests/test_nemotron_ocr_v2_nightly.py | Expands pyproject assertions to cover stable ranges in [local], nightly exclusion constraints in [dependency-groups].dev, and absence of TestPyPI index — good coverage of the dependency-range change. |
| ci/tests/test_huggingface_release_workflow.py | Adds two tests verifying nightly base versions per matrix entry and that nightly_build_publish.py defaults to PyPI — straightforward regression guards for the workflow changes. |
| nemo_retriever/README.md | Adds a short --pre opt-in snippet for users who want nightly Nemotron packages after installing nemo-retriever[local]. |
Sequence Diagram
sequenceDiagram
participant GH as GitHub Actions (schedule / dispatch)
participant WF as nightly workflow
participant Script as nightly_build_publish.py
participant PyPI as PyPI (upload.pypi.org)
GH->>WF: trigger (cron or workflow_dispatch)
WF->>WF: resolve upload_to (default: pypi)
WF->>Script: python ci/scripts/nightly_build_publish.py --repository-url https://upload.pypi.org/legacy/ --token-env PYPI_API_TOKEN
Script->>Script: clone HF repo, patch version
Script->>Script: build wheel
Script->>Script: _twine_upload sets TWINE_PASSWORD in env
Script->>PyPI: twine upload (token via TWINE_PASSWORD)
PyPI-->>Script: 200 OK
Script-->>WF: exit 0
Reviews (2): Last reviewed commit: "Address PyPI nightly workflow feedback" | Re-trigger Greptile
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Move public nightly publishing off TestPyPI and onto PyPI for
nemo-retrieverand the HF-derivednemotron-*packages, while leaving NVIDIA internal Artifactory publishing paths unchanged.This PR updates the public nightly workflows and builder defaults to publish to PyPI with
PYPI_API_TOKEN, removes public TestPyPI routing, and bumps HF nightly base versions so prerelease wheels sort after current stable releases.For local inference installs, the published wheel now exposes only the standard
localextra. That extra enumerates the full local stack and uses stable Nemotron release ranges up to the next major version. The formerlocal-baseandlocal-nightlyextras are removed. Repository uv developer installs still get the latest Nemotron prerelease packages through the uv-onlydevdependency group, which includesnemo_retriever[all],nemo_retriever[dev], and the explicitnemotron-*nightly constraints.For users who install from published wheels,
nemo-retriever[local]resolves stable Nemotron packages by default;--preremains the opt-in escape hatch for prerelease/nightly Nemotron packages from PyPI. The README now documents that path near the local install instructions.Validation performed:
uv run --project nemo_retriever --no-dev --with pytest --with packaging python -m pytest ci/tests/test_huggingface_release_workflow.py nemo_retriever/tests/test_nemotron_ocr_v2_nightly.py nemo_retriever/tests/test_ci_workflows.py -q(33 passed)uv lock --checkuv tree --only-group dev --lockedconfirmed uv dev installs includenemo-retriever[all, dev]plus the nightlynemotron-*packages.local-baseorlocal-nightlyextras and stable-rangeRequires-Distentries forlocal.local-nightly,uv-local, or TestPyPI install instructions outside negative assertions.nemo-retrievernightly publish from this branch: https://github.com/NVIDIA/NeMo-Retriever/actions/runs/27164263927Checklist